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SELECTING INSTRUCTIONAL ACTIVITIES: 
A POLICY-CAPTURINC^ANALYSIS 



, Robert J. Ying'er,. Christopher M. Clark, and Merlyn M. Mondoll 

Considerable activity and interest hasbeen stiniilated in research' 



on teacher thinking by the report of the Panel on Teaching as Clinical 
Information Processing of the National Institute of Education Con- 



erence on Studies in Teaching (NIE, Note 1). To date, research on • 

~\ \' ' ■ ' f -, 

teacher thinking has been direct'ed at four topics: teachers,' planning, 
judgment, interactive decision making, and implicit theories or per-' • 
spectives (Clark & Yinger, i977; Shavelson, Note 2).- This .paper reports 
the results of a study of teachers^ judgments while selecting instruc- 



tional activities. 

■ . ' 

The most' frequei^tly used metho'd of studying and repreaenting judg- 

\ ' ' : ^ ' 

ment processes is policy capturing (Slovic'& Lichtenstein, 1971; ^hulman 
& Elstein, 1975). This\ approach begins with a Simple (usually linear) 
model and attempts to reproduce the inferential responses of a parti-* 
cular judge. Of central interest in this 'paradigm is how judges weigh, 
and combine information in\the form of disc^rnable cues or features of 
the objects' to be judged. ^ ^ . ' . < 

The policy-capturing approach has been used' in studies of teacher 
judgments as theyrelate to characteristics of effective teachers 



^Robert Yinger is a former IRT senior researcher with the Teacher 
Planning Project and an assistant professor of education at the Ug^ver- 
sity of Cincinnati. Christopher ^Clark is the coordinator of IRT's Teacher 
Planirtng Project arid an assocfate professor of education at MSU» * 
Merlyh Mondol is an associate professor of psychology at Saginaw Vi^ley 
State College, specializing in learning and cognition. The authors wish 
to express their appreciatioti to Susan K» Bowns,. Who provided valuable 
assistance in the data collection for' this study. 



(Anderson, 1977)^^cla^^opm or\ganlzatlon (Bor;:^o, Note 3), classroom manage- 
ment (Cone, Note ATr^tlstructlonal strategies (Russo, Note 5), and^lnstruc- 
tional content (Floden, Porter, Schmidt, Freeman, & SchwijLle> 1981). The 
present study adds to this list teacher judgments pertaining to selection 
of l^istructional activities. We have advocated elsewhere (Clark & Yinger^ 
<.-1977)- that a greater number and variety of studies are required about 
teacher judgment of students, about curriculum materials, and about other 
important aspects of the classroom environment before such research will 
be useful in policy and training decisions. The present study adds to ' 
that data by investigating teacher judgment in realistically compl^^x 

situatipns. By applying policy-capturing methodology to judgment situa- 

» • 

tions like those regularly encountered by^elementary school teachers, we 

are also evaluating the usefulness of this methodology for describing the 

« ♦ • ■ ' . * * * 

complexities and subtleties of teachers' mental lives. 

t- * . a 
\ - ' ' -* ' ^ - 

This study of teacher .judgment xis one part in a* series investigating 

teacher judgment during the selection of instructionai^-^iaterials. .This 

series included a study that identifies factors influencing the selection^ 

of instructional activities (Clark, Yinger, & Wildfpng, .Note 6), a policy^ 

capturing study of .teacher judgment (reported here)^ a process-tracing^ 

study. of teacher judgment (forthcoming), a feature analysis of preferred- 
instructional activities ( reported h^e) , and an analysis of teacher's 

self-reported judgment processes (forthcoming). 

'The underlying hypothesis of these studies is that the selection of 

attractive, appropriate, and effective instructional activities is an 

important step in teacher planning" for instruction (Yinger , Note 7). 

Furthermore, teacher judgment is>»an important psychological process in 

this activity-selection task. 



These are the major research questions guiding this* study* 

' !• What factors do teachers take Into account in selecting instruc- 
tional activities? i 

2. How do teachers differ in- the relative empTiasis they place on 
factors influential in selecting instructicgial activities? 

3» To what extent Is the policy-capturing lapproach adequate for 
representing teacher judgments about »(iomplex instructionalr 
activity description^? ' ^ \ , 

^ , 4* What do teachets' judgments reveal about wh^t coilstitut^es a good , 
* Learning activity? . " ' , 

♦ , ^ . - . / Method ' 

Subjects . ^ . . ^ ^ . ^ 

Nineteen fourth-and fiftH^grade teachers (five males "and 14 females ,in 

thelT late 20s to middle 50s) from two Michigan schooi>districts voluntTeerfed 

(and were paid) to participat^e in the study* The average teaching exper- 

ien^ewas 13 years, witha range from five to 33- years*' Twelve teachers 

taught in self-contained classrooms, while seven taught in team- teaching 

Situa^tions or a combination of 'te&m- teaching' and departmental arrangements. 

Seven c^f the^ t-^achers tatight ift urban settings, 11 in suburban communities, 

and one in a rural area. Although all the. teachers were taking graduate 

* 

courses, only nine of them were in master's degree programs* The rest 
jwere taking graduate courses in order -to be peryianen^ly certified as 
j:eachers* ^ ^ • ' * ^ . 

Materials . * - ' , ' 

Thirty-two one-or two-page descriptions of language arts -writing 
activities were derived from a*?:tivities selected from a commercially 
available instructional catalog of language arts activities for upper 
elementary classrooms (Forte, Frank, & McKenzie, * 1^73) * the activity 
descriptions, w^re all presented in the same general format: an activity 



title, a Qne-or two-sentence statement of the purpose of the activity, and 
a ^^li^ting of the steps^ involved in planning and conducting the activity. 

Each activity description , was edited to reflect five importajit dimen- 
sions teachers use td jijdge the quality of language arts instructional 
materials (Clark, Yinger, & Wildfong, Note 6). These dimensions, or cues, 
were (1) student involvement, (2) difficulty for students, (3) integration 
with other skills or subject matter, (4) demand on teachers, and (5) fit 
between stated purpose and instructional process. 

The 32 descriptions were constructed to represent a full factoral . 
matrix of high and low values for each cue. Four researchers . independently 
rated each activity, negotiating when their ratings disagreed. 

We hypothesized that the evaluation and selection of instructional 

' ' V . 

materials Is not' one, but a series of judgments leading to a final, decision 

to imp-lemen^t or not implement an activity. To investigate this process and 

to better reflect the complexity* cr| f he* judgment task, each participant re- 

'\ ^ ^ 

sponded on a nine-point continuum to four questions about each actiyit^.. 

' » ' 

1. How 'attractive is this activity to you? ' " 

^ ' , , ' , 

2. How appropriate is tliis activity as part of a catalog of language 
arts activities- for fourth- and fifth-grade teachers? 

3. How likely would you be to' use this activity as it is in your 
present classroom? ^ ^ ' • 

4. How effective do you think this activity .would be for your students? 
These four questions, in effect, represent" our hyj^othet model of 

. the activity judgment process. In selecting materials for clas^oom use, 
(e.g., while browsing through a collection of instructional activities) 
a teacher may first be personally attracted to an activity, ' then assess its 
appropriateness fot the content and purpose (s)he has in mind, and finally 
evaluate/ its potential value for his/her own students. By phrasing the 



foAir judgment questions relative 'to different reference groups (I.e., In 
reference to the teacher^, to fourth- and flfth-?grade teachers in gQneral 
and to .the teacher's own class), we hoped' that the components of the^selec- 
tion process might be" more- independently visible in the data, * 

Procedur e ^ , • ~ . * . 

. After a set of six warm-up activi4:ies and a question* and answer session 
each teacher was given a loo^e-leaf booklet. containing the 32 activity des- 
criptions, which were arranged in a different random order far each teacher. 

» * 

The teachers proceeded through the booklet of activities, responding to 

the four judgment questionslDn the reverse side of , each activity description 

Data Analysis \ - ' . . 1 . 

The study proposed to assess tfie degree to which judgments about in- 
structional materials could be modeled by linear mathematical represqnta- - 
tion^. Simple linear regression equations have biaen shown' to provide, in 
a number of cases, very good explanations of judgmental responses (e.g., 
Goldberg, 1968). Therefore, we computed linear regression equations for 
each judge (teacher). * The five oues'were treated as independent variables 
"pn which the ratings given to each case were regressed* Using programs 
from the Statistical Package for the Social Sciences, 76 regression equa- 
tions were computed. ^ . ' ' * 

The regression equations produced by the analysis were ^amined in 
.two ways. ^First, regression, weights wdre studied to determine which factors 
were significant predictors of each judge's rating of the in8t;rucjt:ional ^ 
activiti^. Squared multiple correlations were used to assess the extent, 
to which a judge's rating was systematically related to the 'five activity * 
features yaried across the cases. ThougV judgment policy analysis is most 



powerful as an Intra- Individual method, a second analysis was conducted on 
the mean weights of- the r,egresslon' models for each, of the four judgments 
to examine any group trends* ^ 

. • Results 

Individual Judgjaent Pollcles- v 

The analysis of ^acher judgment produced* four judgment models (re- 
gression equations) fo^'eack/^ the 19 .teachers. Since space limitations" 
do not permit a presentation of all ^ 76 of 'these models,^ one judge was 
selected for Illustrative purposes. ^ ' * <r 

Teacher 2 Is a f Ifth-'gt^ade teacher- In an urban school j^lth 16 ye^rs 

of experience.. Xhe mean rating glVen by this judge to all activity des- 
^ *. ' • • « 

crlptlCns was 6.22 with a standard deviation of 2.66. T^ average squared 
multiple correlation from the uegr^sslon analysis of the^ four judgment 
models was/. 40. (adjusted*s2_=^28)^'whlch;indlcates that, as a conservative 
estimate^ a little less than one-thlrd^of the variations* In Teacher 2's 
rating can be accounted for by the activity features .studied. 

•For the portion of thi-s teacher's. judgment accounted for by the re- 
gression, analysis, a fairly consistent weighting pattern is present across 
the four judgments. In Table 1, on^ can see tha,t Teacher 2' 9 judgments of 
the activity deS'criptions were influenced most by the perceived difficulty 
of the activity for the students (Cue ^: difficulty) and the intejr^tion 
of various lajiguage arts skills or other subject mattet in the activity 

(Cue 3: inteyration) . Also-, for three of the four judgments, the fit of 

f 

the' stated purpose of . the activity with the described instructional process 
(Cue' 5: fit) was a significant influence.. The potential student 'involve- 
ment in an activity and the demand -that an activity puts oxf the teacher did 



not reliably influence this teacher's rating's.' From this description 
bne might predict that Teacher 2 is Tikely to search for and use language 
arts activities that are* well organized and constructed, .that are not too 
diffi^cult for the^ students, and that .integrate a variety of language arts 
skills' or^ other subjeQt matter. 

' - ' T^le 1 

•^-Regression Weights for Four Judgments 
^ Determined by the Judgment Policy of Teapher 2 



Judgment 



Feature 

: 1 


1 

Attrac- 
. tiveness 


2 

Appropri- 
■ .ateness 


; 3, 

Likelihood 
J, of us^^ 


4 

• Effec-^ 
tiveness. 


Cue 




1 ' — 






^. Invplvem^nt 


,.13 • 


.00 


.44 ; 




2. Difficulty 


-.81** ' 


-.59* 


-1.63*** 
t 


-1.34*'** 


3. Integration - 


• .63** 


.59* ' 


.87** 


.66* 










4. Demand 


^.13 '. 


- • .22 


•;3 


•'-.31 


5. Fit 


.63**.' • 


.47 


• ^.75* 


.72* 



Note. *P<.10, *^<.05, ***;P<.01. 



Composite Judgment Policies 

Teacher 2, with judgni^nt models cljaracterized by moderate consistency 
and three significant regression ^^^ights,. was quite , different from the 
«st of the judges in this study. Across all four judgment tdsks, 4^% 
of the regression equations had no significant regression weights, and a 



^ ' ■ . ' ' ' ■ - \. : ■ • ' > 

majority of the significant regression" mftjdels had. low squared multiple 
correlations (nreai\ r2 = ,18; range *from,^; 03 - \ 50^ • y. 

To more closely examine .the, composite- jud^ent policies, average- 
regression weights were computed for the five features that were mani- 
pulated in -the activity descriptions* (see T<^le 2).^ For activity attrac- 
tiveness judgments, the feafur,e most heavily weighted was Fit. This was 
closely followed by the ^eatur,e; DeTnand. Difficulty was,' on the^average^ 
negatively weighted, that is, activities perceived as being too difficult 
for the students w^re judged as being less attractive. * 

Table Z ' . „ . 

. , .Mean Regression Weights for Each Feature Cue - 

• ' .1 * Judgment Task 



Feature 



' 1 

Attrac- 
tiveness 



Appropri- 
ateness 



• Likelihood 
of^usd 



Effec- 
tiveness 



Cue 

'1. Involvement 

2. Difficulty 

3. Integration 

4. Demand 

5. Fit "* 



.07 


.02. 


.11 ' ~ 


X -62 


■(.40) 


(.31) < ' 


•C.A3). V 


'(.40> 


-.16 


-.04 


*-.-.14. 


-.i7 


(.34) 


(.28) ' . - 


(.52) 


(.43) 


.11 


.01" 


.-15 


.05 


^(.50) 


(.35) 


. (.53).' v' 


V (.42) 


.21' 


.15 


.11. 


.12 


(. 42) 


'(.35) 


(.48) , ' 


. C.3'8) 


'.24 


.21 ..' 


.23 


.28 


(^^2) 


> (.22)- 


. (.35) ■ 


(.36) 



Note.* Nbnb*ers in ^rentheses are standard deviatiotis 



11 



The average weightings for tfie second judgment task, that of Approprla 
teness, followed a similar pattern to that of the flr^st judgment. The high 
est average-weighting was given to Fit followed by the weighting given to 
Demand* For this 'judgment, however,. Difficulty had a negligible negatfve, 
weighting • . ^ 

The composite policies for the third and fourth judgment tasks, those 

. ' ' ' " , . * 1 . * 

of likelihood of use and effectiveness, followed the pkttBrn of tHe first . 

two ^Judgments. In both casea, teachers gave Fit the^ highest average weight 

and Difficulty a negative weight." * ' * , 

, '•' ' i . 

Characteristic Judgment Policies ' . ^ ' 

The' ^omposlte judgment policies 'desci?tbed lii the previous section 
provide some notion of the tendencies among thefee teachers for weighting 
certain activity features more highly than others. The fact that'tfie 
regr.esslon weights lif Table 2 are averages, however obscures and often 
cancfe^s out' individual .differences amortg the feachers.' The large extent 
of these differences is reflected in the large standard deviations in- 
dicated in Table 2. ■ . ' . 

^ "To further analyze similarities and differences among the teachers,^, 
the standardized regression coefficients for the A3 significant policy 
equations (£.<. 10) were mapped onto polar cooifdinate graphs (Figures l-Aj. 
These graphs allow a figural representation ^nd comparison of th^e signiii- 
cant judgment models for each judgment task. ^ The heavy solid pentagon 
represents^ a weighting of zero for each cue, and positive and negative ' ^ 
.weightings are^reprdsented by'points outside and .inside of pentagon,* 
respectively. Sinpple models (Having only one significant regjcession ' 

weight) are indicated by narrow solid lines. Since for "these models all 

\- ^ . / 

•^ther Weightings are tteated as zero,* the three sides not tatiggnt to the 




Figure 3. Representation of Irl 'significant policy equations 
for Judgment Task 3: Likelihood of use. 



r 
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weighted cue are representied by the heavy solid lines. Complex models 
(having more than one cue weighted) are indicated in these figues by 
textured lines, • ^ 

A variety of information can be obtained from thfese figures. For 
instance, Figure 1 represents the 12 significant models for the f-irst 
judgment task^ * attractiveness. Of the 12 models, 10 are simple, relying * 
on only one cue. Each of the five cues manipulated in the model has 
been significantly weighted by tl^e teachejrsj but the frequency of weightings 
varies from only one weighting (for Fit) to five weightings (for Integration) 
Fit arid Demand were only weighted positively. Difficulty was only weighted 
negatively, and Involven^ent and Integration were weighted ^oth positively 
and negatively. Similar information can be obtained for the other three 
judgment tasks, and in addition, the point labels allow comparison of the 
same teacher's judgments across alt four judgment task^. (Only four of the * 
19 teachers had no significant equation for any judgment -task and therefor^ 
do not appear on any figure. Two teachers appear on only one figure .0 

Since,. with the exception of .Teacher 2, the models Represented in 
Figure's 1-4 have low r2 values, these graphs may be best used for com- 
parative purposes, indicating the wide degree of individual differences 
among the teachers. They do, 'however, suggest some interesting trends in ^ 
the weighting patte'^^ of the teachers who are represented. 

Across the* four judgment tasks certain cues were significantly weighted^ 
much more often than others. One can see in Table 3, for instance, that 
Integration was significantly weighted 16 times, while Fit was significantly 

^ r 

weighted only five, times. , The number of significant weightings for the 
other three cues fell ^somewhere in between. It can also be obfiferved that 
for most of the feature cues, the number of significant regression weights 



was fairly, consistent across t^e f our . judgment/ tasks. The most obvious 

^ • '' ^ ' ^ ' 

deviation f^rom this trend -1^ for Fit^ where three of the six significant 

/ ^ ./ 

weights assigned to the judgment tasks wer^ given for the fourth judgmi^nt 
task (effectiveness).. 

\ 

Table 3 

^ - * Significant Regression Weights'"^ 
. for the Five Activity Featury^s in Each of the Jour 










Judgment /Tasks C 

' /. ■■ / 




* t 






/• / 

^---nJAidgffient^Tasl^^ 




Feature 

/ 


J 




12 3 

Attrac- Appropri- Likelihood 
tiveness ' ateness. ' of Use 


4 

Effec- 
tiveness 



Cue 

1. Involvement 
,2. Difficult;: 

3. Ir^tegration 
. 4. Demand 
• 5. Fit ' 



3 
3 

i 

5 
3 
1 



3 
1 
3 
4 
0 



2 

4~ 
4 
4 
1 



Other trends in weighting patterns a'cross the four judgn 



2 
4 
4 
2 
3 



ent tasks re- 
late to variations in positive and negative weights assigned to the cues. 

^ %Fit between tl^e stated purpose of -an activity and the described instructional 
• . ~ ' ^ , ■ , ^ ^ r 

process was only assigned positive weightdngs. ' This, indicates the higher 

rating of activities having a close fit between its purpose aijd instructional 



process. 



\ 



is 



• ■ , ■ . .' ' ' 1 

\ 

One-sided weighting was also apparent for Difficulty. No teacher slg- 
nlffcantly weighted this feature positively. This fits expectations that 
an activity judged as being too difficult would not be rated highly. Note 
t^at of the five teachers *slgnlf leant ly weighting this cue, only one teacher 
used It In tne second judgment task, appropriateness of Including the acti- 

• • « ■ . In 

• vlty in a set of activities for teachers of fourth or fifth grades. 

liivolvement. Demand, and Integration, all had both positive aAd nega-* 
tlve weights assigned to them. One might predict that an activity perceived 
as having greater student \nvolvment would always be rated more highly than 
one having low student Involvement, but Figures 1-4 show t;hat two of the 
teachers gave significantly l^igh ratings to activities with low student^ 
involvement. One might also predict that activities judged as being de- 

• mandlng on the teacher would not be given high, r^atlngs. This is not always 
<Lrue, as Figures 1-4 indicate.^ Five teachers weJ^ghted Qemand- in a signifi- 
cantly positive* manner on at least one o-f the four judgment tasks,^....^nd only 

> one. teacher weighted Demand negatively. ^ ^ . \ 

V 

The degree t6 which other skills or subject matter^ are* integrated into 
'an activity appeared to be an important activity feature for many of these- 
' teachers. Integration had the greatest number of significant regression 
weights assigned to it and was weighted negatively as well^ as positively. 




One might predict that actlvWles higher ^oiulntegratlon wouJ.d be weighted 

. 

higher. This was true for the weightings, of five teachers. But thrtee 
teachers assigned negative weight to Integration at, least once, suggesting, 
a preference for activities t^at focus on one skill or subject-matter area. 

In summary,- the simple linear model used here has failed to; capture 
much of the richness and compTexlty of these teachers' judgments. In those 
instances where regression equations had any slgn;iflcant descriptive power, 
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the large individual differences in the models suggest the presence of 

highly idiosyncratic processes for the select:^^^ ^of instructional activities. 

% 

Activity Analysis 



thus far*, we haye been d^lscussing what this, study reveals about tire 
various factors inflyencing Che ju^ments, of these teacjiers. What do their 
judgments of the activities suggest about which characteristics of an 
activity make it especially attractive, appropriate, useful, or effective? 

Since the four judgments that the teachers were asked to malce ab6ut 
each activity were highly inteircorrelated (mean r = *82), we averaged the 
ratings giren>by all they teachers for each judgment related to each^ activity 
to produce a general ii;<dicatiqn of quali.ty. ^ The mean rating given to the 
activities was 6.05, with, a range from 3.72 for the lowest-rated activity 
to 7.55 f^'the highest rated activity. 

Table 4 lists the characteristics of those activities rated highest * 
(in' the top 25%). and those rated lowest (in thfe bottom 25%). Since the 32 
activity descriptions were constructed to represent 'a full factoral matrix 
of high and low values' for each of the five activity features, there is no 
repetition or replication of any one'activity profile. Therefidre, at best, ^ 
the data presented in Table 4 can only suggest the contributing effect^ of 
individua'l activity features. (; 

Hs,^ For the most part, the data in Table 4 reveal no consistent trends for 
activities in each grouping.' The only interesting exceptions .appear to be ' 
the features Demand and Pit for the bottom .group^ All but one of the 
activities in the bottom group were low in Fit and all but two of the 
activities in^this group were low in Demand. 



In activities of this complexity *it is likely that various features*. ^ 
counterbalance each other in some configural manner. This seems to be 
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Table 4 



Characteristics of Activities in\he Top 
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supported by"" the data' ia Table 4 so that no^eatute was unanimously, high *or 

low across the two groups* For Instance^ the activities In the top group 

' // . \ 

'were not all high in Involvement or high, in Integratipn; nor were the ' 
activities ;in the bottom group^^^all low in Fit. ^^:t is likely that teach'ers 
use policies containing ^tradeoff s .like "this ^activity shofild be worthwhile 
even though it may demand a lot from me, because^ It integrates several im~ 
portant language arts skills." . ' ^- • . 

A difficulty in assessing *which features pf tfi^ activity jdescrip^tions 
contribute most strongly to high^or low rankings is related to our method. 

* ' ' 9 

We chose to control only five features of the activities knowing that there 

are many more sources of information in aativitiel^ of this type. For J.n~ 

\^ ' ' ' ' ' . \ 

stanre, in a previous study (Clark, Yinger, & Wildfong, Note 2) we, found 

. \^'^, - M ' 

tKat teachers mentioned an average of eigHt^^^a^^^^c^^ .features and as . many 

* " ' ' -/j v. ,} ' 

as 11 when making jydgtaeitts of this type^C?X=r ' > ' V 

.'-'^ 

^ Discussion , ^ . 

What can be learned from this study about t^^chersi selection of in-^ 
structional activities? The f^r research ^questions posdd earlier will 
provide a framework f6r our answer. , . 

What Factors Do Teachers Take into Account ' . » 

in Selectifig Instructional Materials? - . 

, «' ^ , 

Teachers seem to be taking a large number of -factors into account 

\ > - 

in their judgment^ about instructional materialdv* .The factors or cues that 

we selected to systematically vary^ in this study did- not fully account for 

the* richness^ of the teachers judgments. On*^ the average, less ttian one- . 

* a * * * 

fifth of the variance in teachers^ judgments^Vere accounted for by the model 
used here (although in several cases one- third to one-half of the variance 

2d. V- . ' 
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was accounted for). This Is not to sav, however, that - the feature cues I 
selected for use In this study were not being used by \it±s group of teachers. 
Fifteen of the^ 19 participating teachers <ised 9ne or more of the five cues 
In, a significant manner* For this sample of teach.ers the f^ttors most fre- 
quently Influential In , their'' judgments were Intbgratioriy Difficulty y and 
Demanct, . * I , 

How Do Teachers Differ In the Relative EmphaslsNThey Place 
on' Factors Influential In Selecting Instructional Activities? 

^ .The wide variations found In cue usb are striking^ These wide variations 



made It vlrually Impossible to say anything meanlnigful about average or'.group 
judgment policies, (See for Instance the large standard deviations In Table 
2.) . By focusing on tfiose \eachers who had significant policy , equations, w^ ' 
were ^ble to get a somewhat better Idea- about the ways In which they were' 
weighting the five feature cues manipulated In^ the study, ^Figures *l-4 pro- . 
vide a much more accuratve'.plctui;e.of the variety of Individual judgment 
policies • " " • ' , 

Based on the^e.flgues It is also possible to say "a. few things about ' 



trends in cue use. For instance, across the four 'judgment^ Integration was 
Incorporated most frequently. Also, the we-ightlng for Integration was 
negative as well as positive. Three teachers tended to ju^e activities 
more favorably if they^ did jaat. Incorporate other language arts skilly, 
while five different teachers favoted more Integrated activities. Similar 
variety was found for Involvement and Demand* However, when ^it was used 
it, was only used positively. Difficulty was used only negatively, indi- 
cating a lower- rating of activities that are perceived as being too difficult. 
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To What Extent Is the gollcy-capturlng Approach Adequate 
for -Representing Teacher Judgments about Complex Instruct 
tlopal Activity Descriptions ? ^ ' ' ^ 



Based 6n the Results of this study, policy-capturing methods seem inade- 
quate fx>r modeling a task of this complexity. The low number of signifi- 
*cant policy equations and the small amount of variance that could be 
accounted for'attest to some kind of mismatch. One might argue that the 
fatij/^ .s^->iot in the method but in hbw- it was used, but we feej. that 
understanding Hiujnan judgment in-'contaxts as complex as 'this presents 
several insurmountably l^rriers to the use of policy-capturing methods^ 
Jhe first barrier ' is the trade-off that one needs to make betweeTi 
task validity (ecological validity) and the limits of the linear modelv- 
The linear model fun^ioris most effectively in situations in which judg- 
sent stimuli are^j^yeSented in a fixlly factorial design with cue values 

represented in an unambiguous manner. As the information presented to a 

^ P \ ^ • ^ ■ . - " 

iud^e becomes richer ( in ^ terms of information content) the ability of the 

^ f ' * 

linear model to describe the judgment process drops. This decline is 

largeljT'ciue to the additional, uncontrolled information available in com- ^ 

pl6x situations that can influence judgments.' The trad^e-of'f between task 

complexity and descriptive* effectiveness has been shown repeatedly in 

studies of teache^ judgment. For example, in studies of teacher judgment 

.using simplified summaries or profiles of student or curricular charadter- 

istics .(e.g. , Anderson, 19^7; Shavfelson, Caldwell & ^Izu, 1977), the linear 

model has much more 6f fjectivel^ captured judgments than in mor^ rec^ent 

' .... ' • 

studies that have attempted to use more complex and realistic' ob^^cts of 

•judgment (e.g.,^Smith & Glass, Note 8; Byers & Evans^ No£e 9;. Floden et al. 

1981). These results, when combined with those of this ptudy, suggest that 
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policy capturinjg can only be used effectively In judgment task^'in which 

' * 

the cues can be represented in a simple, unambiguous manner without strip-' 

ping away thevrichness and complexity of the real task. 

A second barrier is related to the assumptions underlying the- linear 

^mpdel. By viewing judgment as a fairly simple additive or multiplicative 
* - 

^ process where discfete .features combine simply with other features;, we' 
*may be excluding the more holistic and eo'nfigural comppnents in judgment/ 
, It may be, that judgment research needs to attempt to' accoimnodate more 
complex cognitive structures such as schema^, .prototypes, and scripts. ^ 

A- rfelated Assumption, of the- linear model and policy capturing models^ 

' • ^ ' / 

is that the cue is ^xerceived exactly the way it is presented.- , Recent re-' 

« 

search in perce'^tion: has revealed mucli aboqt the impT)rtance of context and 

• ^ 

the knowledge and experience that the' percetver brings to the situatiori. / 
One cannot be sure that what is presented is the same ^s what is j.udged. 
Judgment is as m\ich a* perceptual task as It is a conceptual task. 

Anotjier limitation inherent in the policy-capturing method is the loss 
of^ infonnatioif about judgments when summari|;Lng across instances. As judg- 
ments are average^, the^^unique factors influencing each judgment are lost. 
Tkfs is especially true wiien researchers try to represent *1t:he 3udgments of 
a group. of people. ^ , r* • . ^ 

What Do Teachers' Judgments Tell about VHiat ^, 
Constitutes a Good Learning Activity ? ' ^ , f 

Analysis of th^ activities tlie teachers rated highest and lowest re~ 

*\ . - 

vealed no consistent trends for activities in each grouping. As was pre- 



viously menti9iled^ this is most likely .due to artificially resttlcting the 
number of cues studied. Tl\ere was a trend for teachers to reject language 
arts activities that were demanding for the teacher and In which the stated 

• N - 
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purposes of the activity did' nbt fit well with the^pVocesses prescribed icv 
students. Our comparative. analysis of the actlVltlels rated in the highest 
and lowest ^quar tiles indicates that no single activity feature (of' the fiV^e - 
manipulated) was always ^pre^sent among the highly rated'*' ^ctlvlfcl^s or Always- 
absent among the activities rated lowest. This suggests that the teachers ' 
may have been using a conflguiiial process in judging the attractiveness^^, 
likelihood of use, .and^ potential effectiveness of language arts activities. 

• ' . Conclusions 
Teacher judgment. play§ an important parfdn teacher selection of in- 
structional activities. The policy-capturing Analysis presented in this 
study shows that teacher judgntents of the attractiveness, approprlat^ess, 
usefulness, and potential effectiveness of language arts activities are 
highly intercorrelated (i.e., that thera are tWo kinds ^of activities: 

a/ 

those rated high and those rated low) yet there are striking individual 
differences among teachers in which' activities they rate high or low,^ 
and why. 'Furthermore, the relatively loy^^e^^^ptive p&wer of*a , 
five-cSe model of teacher judgment suggel^^i^t, in a realistically com- 
plex judgment task,* teachers use a wider variety of information, possibly 
in a more configural fashion^ than can be adequately represented by a linear*^ 
regression model.** 

In an attempt to move beyond the limitations of the linear .model in 

* 

describing complex human judgments, we have conducted a process-tracing * 
Study of teacher judgments about thi^f sape language 4rts activities used in 
the presont stlidy. In the process tracing approach, the teacher is askJL 

to think aloud vhen coming to. a judgment. Transcripts of the teacher's 

■* 

deliberations are then analyzed ta trace the prpcess whereby (s)he arrived 
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it a decision. Data analysis Is still In progress, but«prellmlnary results 
suggest that the process-tracing approach provides a more appropriately 
rich, complex, and conflgural picture of human judgment In realistically 
complex situations than does the policy-capturing approach. 

This study dealt with one aspect of how curriculum materials, Instruc- 
tlonal^nsiderations, and human perceptions 'come together in the'mlnds of 
teachers as they decide what to teach and how to teach it. We sought to 

■ ■ ■ ■ % 

understand why and how these Important decisions are made to' better under- 
«tand how the content and process of instruction come to be. Teachers' 
judgment processes are complex and differ among (teachers. We recommend 
that others use more appropriate, complex methods of studying teacher judg- 
ment in action. 

9 
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